Repository navigation
KSES: Preserve escaped shortcode brackets in text nodes - #14770
salvatorecorsi wants to merge 3 commits into
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
|
Test using WordPress PlaygroundThe changes in this pull request can previewed and tested using a WordPress Playground instance. WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser. Some things to be aware of
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation. |
|
@salvatorecorsi this is an issue we worked through when building the refactor. there are a lot of shaky ways to try and prevent the unescaping of square brackets, but ultimately it seemed best to leave it, for a number of reasons:
I considered something like you show for text nodes, because those don’t currently have any processing applied. as you scope-limited this to text nodes, that’s nice, but also sets up a weird expectation where some escaped brackets are preserved and others aren’t. by the way, I think it’s possible to speed up the detection of these inside text spans. while we might generally have some issues using a PCRE pattern to find character references, in specific cases we should be able to find comprehensive matches. $bracket_pattern = <<<PCRE
~
\A # immediately following the found &
(?:
(?:[lr](sqb|brack)) # named character references
|
#(?:
x0*5[BD] # hex numeric
|
0*9[13] # decimal numeric
)
)
\K # discard the results/don’t allocate
~ix
PCRE;
while ( $at < $end ) {
$next_at = strcspn( $raw_text, '&', $at, $end_of_span );
…
// Filter on candidate matches.
if ( 1 !== preg_match( $bracket_pattern, $raw_text, $match, PREG_OFFSET_CAPTURE, $at ) ) {
$at = $next_at + 1;
continue;
}
// rely on the real decoder just for these, because the semicolon rules are complicated.
$reference = WP_HTML_Decoder::read_character_reference( … );
…
}anyway, that’s just a thought. it could speed up your approach, or it may not. the code I shared is probably terribly buggy, but it’s meant to convey the idea only. for attribute values this breaks down hard because we expect them to be transformed. unless we re-engineered |
|
@dmsnell Thanks for explaining the earlier reasoning. I've investigated this further with Codex-assisted analysis and automated checks, including a comparison against the legacy path and a small performance benchmark. The compatibility case I think is worth preserving is existing entity-encoded text. With a registered
I agree that doubled brackets are the documented shortcode escape. My concern is that they do not retrofit existing encoded examples, which can start executing after a restricted author's save. They also depend on the shortcode being registered: For ordinary text, the mapping uses bracket order, not byte positions. The current newline/NUL normalization and HTML escaping preserve that order; the original token and the same HTML decoder identify which brackets came from references. This is the specific invariant the patch relies on. It does not restore raw HTML or change attribute, URL, or CSS sanitization. I agree that this reasoning cannot simply be extended to attributes: their filtering can remove or rewrite substrings. I do not have a sound general provenance solution for those transforms, and would not reinsert raw attribute values after sanitization. The text-only behavior is consequently partial. Allowed Your performance concern is justified. On PHP 8.3.28 CLI, comparing the exact base and patched KSES sources with warmed, repeated measurements, this synthetic input: '<p>[ ' . str_repeat( '& © © — ', 500 ) . '</p>'takes roughly 3.43 ms before versus 6.89 ms after, with identical output. One literal bracket triggers an unnecessary second decode of all those unrelated entities. Ordinary 9 KB prose measured about 0.71 versus 0.73 ms; these are local function microbenchmarks, not request-level timings. I would first address that avoidable work using a cheap candidate filter followed by the real decoder, as you suggested, and measure it again. Would you consider this limited ordinary-text compatibility fix worthwhile after reducing that cost, or would you prefer a broader approach first? |
|
@salvatorecorsi there’s a lot in your response I unfortunately find hard to read through and understand. but here is perhaps an incomplete and unsatisfactory response. whether we could preserve these in raw text nodes is something we knew we could do. I’ve explored other text node normalization as well. what I’m most worried about in doing this is presenting a situation where some shortcodes are preserved and others aren’t — one aspect of the I appreciate that in your compatibility chart, you note that this is a divergence in behavior for how the old KSES worked. However, I don’t think it tells the full story. It’s an important question, and I want you to know I’m thinking through this with an open mind (and my rambling is characteristic of me trying to share my thought process transparently). there may be opportunities to explore this at the end, using your approach with counting brackets, but instead counting actual shortcodes. that might even work for attribute values, except for some situations. one glaring example are CSS selectors which might appear, for example, in <div data-target="[hidden]">That’s not a Shortcode, though There is vague suggestion that Shortcodes may only appear in text nodes and inside attribute values. That’s vague because the shortcode parser extends beyond that. What do you think about exploring this idea:
|
|
That would give text and attributes the same rule: normalization must not introduce a shortcode that the raw input didn't contain. I'll prototype the comparison at those flush points, including encoded tag names and CSS-selector attributes. The registration-dependent parser is the part I still need to work through. |
|
I tried this in an isolated prototype. It catches encoded names in both text and attributes, and bracket escaping leaves the decoded CSS selector unchanged. I'd keep detection independent of registered tags so plugin activation doesn't change the saved result. Counts still lose which occurrence was literal: There's also |
this is an easy escape hatch, and something I plan on proposing, but it’s a backwards-compatibility break with unknown consequences. shortcode detection is kind of a wild mess.
you are starting to see the complicated cases this touches. another change to shortcodes would free this up, and I think the intent is that shortcodes today only exist within complete text nodes, or inside HTML attributes, but the current shortcode parser doesn’t do that. and this is all work meant to ensure the prevention of creating shortcodes, yet we have to deal with representing non-shortcode lookalikes. a good metric might simply be that we create nothing that could parse as a shortcode through this all brings me back to shortcodes, where the heart of the problem is. we’re supposed to escape anything that could appear like a shortcode, meaning that if we ever expect |
|
You're right that registration-independent detection would change compatibility, so I won't build that assumption into this patch. The prototype only checked the decoded selector value; it didn't establish that every shortcode-processing path remains compatible. I'll keep the implementation unchanged for now, until there's a rule that prevents newly created shortcode syntax without rewriting literal selector values, including when removing markup joins text across tokens. |
Description
The HTML API implementation of KSES decodes escaped square brackets in ordinary text nodes. This changes WordPress shortcode behavior:
do_shortcode( wp_kses_post( '[example]' ) )invokes a registeredexampleshortcode instead of displaying its literal spelling. The same regression affects examples insidecodeandpre, and restricted authors encounter it through the normal post-save filters.Preserve the distinction between literal brackets and brackets decoded from character references. Use the existing HTML decoder to identify reference-origin brackets in the original text token, then retain those as numeric references when serializing the already-normalized text. Literal shortcode syntax, other HTML character decoding, and existing newline/NULL handling continue through the existing paths. A compact byte-string map keeps the additional storage proportional to the number of brackets.
This is a focused follow-up to the escaped-bracket TODO in #13271. It covers ordinary
#textnodes; HTML attributes and atomic RCDATA (title/textarea) need separate provenance handling and are outside this patch. It does not change shortcode parsing or claim that the entire KSES migration is complete.Trac ticket: https://core.trac.wordpress.org/ticket/66208
Reproduction
examplewhose callback returns a fixed marker.[example] [example]withwp_kses_post()under the default WordPress bootstrap.do_shortcode().Before: the callback runs twice. After: the encoded example remains literal and only the intentionally written shortcode runs. The regression tests also exercise
wp_insert_post()as an author withoutunfiltered_html, then read and render the stored content.Validation
Refreshed on October 23 after a normal merge of current trunk (3fb786c), with unchanged source and test hashes. Native PHP 8.3 / PHPUnit 9.6, using the workspace's serialized, isolated test-database wrapper:
npm run buildpasses; built KSES is byte-identical to the changed source.The new static provider covers decimal, hexadecimal, named and alias references; leading zeroes; omitted semicolons; either encoded delimiter; mixed literal and escaped shortcodes; nested/unknown/non-bracket references; repeated sanitization; CRLF; PRE newlines; and NULL normalization. A separate test exercises real restricted-author post saving. The full Core suite and Docker E2E have not been run.
Validation after trunk update (2026-11-02)
Merged trunk
040bdd6b89913c4272bd4fdc61a79e8347cee21dnormally and validated headbacc5403a224c8e4334fc93122dd498e287aa240.Merged current trunk normally and validated the resulting tree.
The existing broader shortcode-compatibility/design discussion remains open.
Use of AI Tools
AI assistance: Yes. Tool: OpenAI Codex. Used for investigation, implementation, regression tests, local validation, and independent automated code reviews. The account owner has not performed a human review of this patch. WordPress maintainer review is required.
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.